Multimodal Foundation Models For Echocardiogram Interpretation

Multimodal deep learning foundation models can learn the relationship between images and text. In the context of medical imaging, mapping images to language concepts reflects the clinical task of diagnostic image interpretation, however current general-purpose foundation models do not perform well in this context because their training corpus have limited medical text and images. To address this challenge and account for the range of cardiac physiology, we leverage 1,032,975 cardiac ultrasound videos and corresponding expert interpretations to develop EchoCLIP, a multimodal foundation model for echocardiography. EchoCLIP displays strong zero-shot (not explicitly trained) performance in cardiac function assessment (external validation left ventricular ejection fraction mean absolute error (MAE) of 7.1%) and identification of implanted intracardiac devices (areas under the curve (AUC) between 0.84 and 0.98 for pacemakers and artificial heart valves). We also developed a long-context variant (EchoCLIP-R) with a custom echocardiography report text tokenizer which can accurately identify unique patients across multiple videos (AUC of 0.86), identify clinical changes such as orthotopic heart transplants (AUC of 0.79) or cardiac surgery (AUC 0.77), and enable robust image-to-text search (mean cross-modal retrieval rank in the top 1% of candidate text reports). These emergent capabilities can be used for preliminary assessment and summarization of echocardiographic findings

A Datalake on CAVATICA – powered by interoperability standards

The ability to provide actionable data to researchers is one of the main goals of the Velsera’s platforms. Here we showcase the integration of data coming from 10+ datasets coming from different consortia and repositories in the CAVATICA Platform using the DRS standard and the GA4GH Passport. The DRS standards provide the ability to integrate several DRS servers, even though these servers are provisioned by different teams in different organizations. At the same time, the GA4GH Passport provides the ability to treat the controlled data with the right authorization level, while leaving the ability to also cater the registered and the open data.We demonstrate how a user can access data from TCGA, TARGET, TCIA, CPTAC, Kids First, INCLUDE, AnVIL, CBTTC, OpenPTBA, TOPMED and UDN all from the same project. The collected data can be accessed by the user with the proper authorization on the CAVATICA platform; they can be used transparently in tasks, which are workflows executions launched from the CAVATICA Platform in a CAVATICA Project, or in Data Studio environment, always part of the CAVATICA Platform, like any other regular file present on the platform.</p